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Introduction to Image Processing 


What is "Image Processing"? 

• (Definition from Wikipedia) “Image Processing is 
any form of information processing for which the 
input is an image, such as photographs or frames 
of video; the output of image processing can be 
either an image or a set of characteristics or 
parameters related to the image. Most image- 
processing techniques involve treating the image 
as a two-dimensional signal and applying standard 
signal processing techniques to it.” 


02/20/08 


Introduction to Image Processing 




Applications of Image Processing 

• Medical Applications (e.g., Cancer Detection, 
Remote and Assisted Surgery) 

• Security Applications (e.g., Face and Fingerprints 
Recognition) 

• Commercial Applications (e.g., Video and 
Photograph Enhancement) 

• Industrial Applications (e.g., Assembly Line 
Manipulation, Visual Inspection) 

• Military Applications (e.g., Missile Guidance) 

• Space Applications (e.g., Remote Sensing, Space 
Robotics) 
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Sequence of 
Image Processing Tasks 


Histogram 
Thresholding, 
Edge Detection, 
Line/Corner Detection, 
Mathematical 
Morphology, Shape 
Measurement, etc. 
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Biological Vision 

• Visual functions Integrated by Brain: 

> Field of View, Focusing Ability, Depth Perception, 

Motion Perception, Color Perception 

RETINA CHOROID SCLEROTIC 

• Different Kinds of Vision/ “See” and “Understand” in Different Ways 

- Human Vision 

o Image Formed on the Retina/ photoreceptors (or rods and cones), produce electrical transmitted to brain via 
optic nerve. 

o 3 kinds of color receptors - blue, greenwish-yellow and red 

o Position of eyes determines degree of peripheral vision; Visual field of 200°; Stereo vision => depth 

- Animal Vision: 

» Dogs: 

o 2 kinds of color receptors - yellow and greenish-blue 
o Visual field of 240° but central binocular field of view =1/2 human’s 

o Optimal dilation of pupil (= camera’s aperture) + reflective layer under retina => Enhanced night vision 
o Lower details sensing (no fovea); Greater sensitivity to motion 

» Snakes: 

o Do not see color 

o Combination of light receptors: rods => low- light fuzzy vision & cones => clear images 

o Underground snakes: smaller eyes/light and dark; above ground: very clear vision and good depth 
perception. Some species (e.g., boas and pythons): pit organs similar to IR goggles. 

» Insects: 

o “Compound eyes: Bees’ eyes made up of 1000’s of lenses, dragon flies 30,000’s 
o Wide field of view, and better motion perception 
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Biological Vision 

Optical Illusions, Illumination, A-priori Knowledge, Domain-Dependent 
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Image Formation - Physics 


Vision uses Light Reflected from the Surrounding World 
to Form an Image .. 
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Image Formation - Optics 
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Digital Images 

• Mathematical Model: Representation of an Image as a Discrete (Intensity) 
Function of Spatial Samples 

- I: (x,y) — — > I(x,y) = Gray Level at Pixel (x,y) 

- Gray Levels = Discrete Values Taken by Intensity Function 

• Pixel (“Picture Element”) = Image Representation of a Basic Volume Element 
in the World 

- Tesselation: Pixel Organization 


Rectangular Triangular Hexagonal 

• Spatial Resolution = Represents Interval Sampling of 2D/3D Space 

• Spectral Resolution = Represents Interval Sampling of Electromagnetic 
Spectrum 

• Radiometric Resolution = Refers to the Number of Digital (Gray) Levels Used 
to Represent the Data 
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Remote Sensing Imaging 
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Examples of Spectral Response Patterns for 4 Different Types of 
Features - Fir Tree , Clear Water , Barley , Granite - White Areas 
Show the Portions of the Spectrum Corresponding to the 7 
Channels of Landsat-Thematic Mapper (TM) 


1 Signal to Noise at Wavelength X : 

(S / N) x = D x p 2 (H/V)i« A, L, 

I Where 

D } : detectivity (measures detector performance quality) 

(3 : instantaneous field of view 
H : flying height of the spacecraft 
V : velocity of the spacecraft 

: spectral bandwidth of the channel (spectral resolution) 
^ L x : spectral radiance of ground feature 

=> Tradeoff between spatial and spectral resolutions, e.g.: 

To maintain the same SNR while improving spatial resolution by 
a factor of 4 (i.e., decreasing /? by a factor of 2), we must degrade 
the spectral resolution by a factor of 4 (i.e., increase A t by a factor of 4) 
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Remote Sensing Imaging (2) 


Instrument 
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Imaging Examples - Photographs 


. 
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Imaging Examples - Medical 



Introduction to Image Processing 
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Earth Sc\er\ce Imaging 

ETM/IKONOS Mosaic of Coastal VA Data 


IKONOS 



Hubble Space Telescope Imaging 

(Barred Spiral Galaxy NGC 16 72) 
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Planetary Imaging 







Image Processing Steps 
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Sequence of 
Image Processing Tasks 


Histogram 
Thresholding, 
Edge Detection, 
Line/Corner Detection, 
Mathematical 
Morphology, Shape 
Measurement, etc. 
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What is Image Processing ? 


Test Case: 

• 1 0 by 1 0 pixels Image 

• 256 gray levels 

• Image = 1 0 x 10 Matrix Made up of Numbers in 
Range [0-255] 
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Image Processing Basics 


Intensity 
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Image Convolution 

Many Image Processing Operations are “Local” 

o Smoothing, Edge Detection, Slope Computation, Wavelet Transforms, etc. 
o Parallel Computations 


Pixel Neighborhood, N 
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Image Convolution 

- Convolution of Image I with Filter h at Pixel (x,y) is defined by: 

I * h (x,y) = I, u , v) , n N I(u,v) . h(x-u,y-v) 

- Theorem / Fourier Transforms (F): 

F (I, * I 2 ) = F(I,) . F(I 2 ) 
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Image Enhancement 
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Histogram Equalization 



Channel: Cray 
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Image Smoothing 




(Edge-Preserving 

Smoothing) 
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Edge Detection 
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Edge Detection 


• Find “Jumps in Intensity”, i.e. pixels where Gradient is Maximum 



• Edge Detection Methods: Compute 1st and 2nd Derivatives 
o Find Maxima of First Derivative 
o Find Zeros of Second Derivative 



Intently Function Along Fir it Derivative of Second Derivative of 

Gradient Direction Above Function Above Function 
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Gradient Operator 

(1st Derivative) 

Sobel Edge Detector (2 masks): 


Gx = 



0 

\ 

f- 2 
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G 
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12 1 / 


Convolution of Gradient Operators with Image: 

• dl/dx (x,y) = Gx * I (x,y) = X U Z V [Gx(u,v) . I(x-u,y-v)] 

• dl/dy (x,y) = Gy * I (x,y) = Z U Z V [Gy(u,v) . I(x-u,y-v)] 

Gradient of image I at Pixel (x,y): 

• Magnitude: GI(x,y) = V(( Gx * I(x,y)) 2 + (Gy * I(x,y)) 2 ) 

• Direction: Arctg(DGI(x,y)) = Gy * I (x,y) / Gx * I (x,y) 

Variants: Prewitt (1 instead of 2), Roberts (2x2 neighborhood) 
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Operators non Isotropic 

o Isotropic Edge Detection with 1, 4 or 8 Masks, 
e.g., Laplacian: 

Introduction to Image Processing 
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Example Gradient Computations 
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Gradient Direction: 

• Gx=508 ; Gy=0 

• Arctg(Gy/Gx) = Arctg(O) = 0 

Normal to Edge => Vertical Edge 


• Edge Pixel = Pixel where Gradient Magnitude is Maximum 

• Can be determined by Thresholding Gradient Magnitude 
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Some Other Edge Detection Methods 




Hueckel 

o Approximate Edges with an “Edge Template 
o S(x,y,c,s,r,b,d) = ( b if cx+sy < p 

b+d else 



Marr & Hildreth 


o Filter Image by Gaussian Filters of Various Variances (i.e., various frequencies) 

o For each variance a , find the zero crossings of the derivative of Image I filtered 
by G oC Q)r I I h .: convolve I with 2nd derivative of G c or Laplacian ) 

o Laplacian approximated by Difference of 2 Gaussian filters 


Canny Edge Detector 

• Designed as an “Optimal Edge Detector” for following 3 criteria: 

» Good Detection 

» Good Localization (more Smoothing improves detection but hurts localization) 
» Single Response per Edge 

• Steps: 

1 . Gaussian Smoothing (Assumes Gaussian Noise) 

2. 2D First Derivative Gradient (e.g., Roberts or Sobel) 

3. Non-Maximal Suppression, i.e. keep Local Maxima in the Direction of the Gradient 

4. Hysteresis via 2 Thresholds, T h and T,; if I <Tj => no-edge; if I>T h => edge; if 
T!<I<T h => kept as an edge only if there is a path to an edge point 
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Edge Detection - Test Image 
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Gradient Thresholding 

Test Image 
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Z « 



Gradient Directions 

Test Image 
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ORIGINAL 


GRADIENT SEVILLE 
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Other Edge Detectors 

Test Image 


Introduction to Image Processing 
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Edge Detection Results 

Boats 
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Fourier and Wavelet Analysis 
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Fourier Analysis 

• Fourier Transform: 

o Decomposition of an Image in a Weighted Sum of Sinusoid Functions 
of Different Frequencies 

o F(I) (x,y) = lfl(u,v) e _Qn ( ux+y y) du dv => Weights 

= lfl(u,v) cos(2 II(ux+yy)) du dv + i|Jl(u,v) sin(2 n( ux+vy)) du dv 
= Real(F(I)) (x,y) + i Complex(F(I)) (x,y) 

{ Amplitude = (Real ) 2 + (Complex ) 2 and 
Phase = Arctg(Complex/Real) 

o Property: Fourier Transform of a Gaussian is a Gaussian 

o No Localization 

• Windowed Fourier Transform: 

— WF(I)(x,y,p,q) = lfl(u,v).G(u-p,v-q) e '^ n ( ux+v y) du dv 

- Gabor Transform when G is a Gaussian function centered at every 
image point 

- Localization but G = Same Envelop for All Frequencies 
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Wavelet Analysis 

• Wavelet Transform: 

— Wav(I)(a,b) = l/V|a|JJl(u,v).W(u-bl,v-b2) dudv 
- W is the “Mother Wavelet” 

Localization (similar to Gabor) 

Better Division of Space(Time)-Frequency Plane : Good for Short- 
Lived HF Components Superposed on Longer-Lived LF parts 
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Discrete Wavelets 


• Orthonormal Basis and Frames 

- Daubechies Wavelets 

- Mallat: Definition of Wavelets from a Scaling Function 


Columns 



( F ) Represents the convolution of Image with Filter F 
(^ 2 ) Represents the Decimation of Image by 2 
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Daubechies Least Asymetric Wavelets 




1,5 
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Example of 2D Wavelet Decomposition 
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Example of 2D Wavelet 
Reconstruction 
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Example: Wavelets Transform 
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Applications Of Wavelets 

• Image Compression 

- Provide a More Compact Representation of an Image 

• Lossy Compression (some visual quality is lost) 

• Lossless Compression 

- JPEG (Joint Photographic Experts Group) and JPEG- 
2000: Lossy Compression 

• JPEG: Compression based on Discrete Cosine Transform (DCT) 

• JPEG 2000: Compression based on Wavelets 

• Image Registration 

• Image Segmentation 

• Image Fusion 
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Image Registration 
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Image Registration 

• If 1 1 (x,y) and I2(x,y) : images or image/map 

Registration = Find the Mapping (f,g) which Transforms II into 12: 
I2(x,y) = g(Il(fx(x,y),fy(x,y)) 

» f : spatial mapping 
» g: radiometric mapping 

• Remote Sensing: 

- Navigation or Model-Based Systematic Correction 

• Orbital, Attitude, Platform/Sensor Geometric Relationship, 

Sensor Characteristics, Earth Model, ... 

- Image Registration or Feature-Based Precision Correction 

• Navigation within a Few Pixels Accuracy 

• Image Registration Using Selected Features (or Control Points) to 
Refine Geo-Location Accuracy 
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Image to Image Registration 

Correlation of Edge Features 
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Image Characteristics 
(Features) Extraction 


Feature 
Matching 




• Multi-Temporal 

Compute 


Image Correlation 

Transform 


• Landmarking 



• Coregistration 
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Image to Map Registration 

Correlation of Edge Features 



Map 
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Application of Wavelets to Image 

Registration 
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Multi -Resolution Wavelet Registration 
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Mathematical Morphology 
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Mathematical Morphology Concept 


Mathematical Morphology (MM) Concept: 

• Nonlinear spatial-based technique that provides a framework. 

• Relies on a partial ordering relation between image pixels. 

• In greyscale imagery, such relation is given by the digital value of 
image pixels 

Greyscale MM Basic Operations: 

Original image 
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Binary Erosion 



l4.0‘ 


: 

* 

* 


Structuring Structuring Structuring 

element element element 


Introduction to I mag e Processing 


Binary Dilation 
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Combined Operations, e.g. 
Erosion + Dilation = Opening 



Introduction to Image Processing 
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Image Segmentation 
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Image Segmentation 

• Image or Region Segmentation is the process that 
generates a spatial description of the image as a set of 
specific parts, regions or objects. 

• Image divided into groups of pixels that are homogenous 
for a given criterion 

o Contrast with surroundings: edge-based segmentation 
Examples: Edge following, line or curve fitting, etc. 

o Similar properties, gray level, color, etc. measured by some 
local statistics such as means, variance, etc.: region-based 

segmentation 

Examples: Region Growing, Region Splitting, Split and Merge, 
Relaxation, Watershed, etc. 

o Remark: Image Classification = Pixel-Based Method (e.g., 
Neural Networks) 

• Segmented Output => Higher-Level Image Interpretation 
Process, part of Computer Vision or Image Understanding. 
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Edge Following 

Prior Processing: Edge Detection and Thresholding => 
Choose Starting Point above Threshold 


Various Methods for Edge Following, e.g.: 

Line by Line Edge Following: Label all Contours 

Give the Same Label to 
all Connected Pixels 


- Contour Following 

1 . Starting Point P 0 with Gradient Magnitude above Threshold 

2. In Neighborhood (4 or 8 pixels) Centered around P 0 , Search in Circular Pattern 
=> 1 st point above Threshold and Gradient Direction Compatible with P 0 

- Graph Traversal 




All Pixels whose Magnitude is above Threshold represent the Nodes of a 
Graph 

Define a Cost Function Based on Gradient Magnitudes and Orientations 
Contour Extraction Performed by Finding Path of Optimum Cost 
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Blood Platelets Extraction 
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Blood Platelets Extraction 


Algorithm: 

1. Histogram 
Equalization 

2. Sobel Edge 
Detection 

3. Dilation 

4. Circular Edge 
Following 
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Line and Shape Detection 

• Build upon Edge Detection results 

- Let us call (G,D) the edge magnitude and direction images of an image I 

- Hough Transform for Line Detection 

• Every line can be represented as: xcos0+ysin0 = p 

• Create an Accumulator: 

- For each (x,y) for which G > threshold, compute p = 
xcosD + ysinD 

- Increment the counter of (p,D) in the accumulator 

- The pair (p,0) corresponding to the maximum counter 
represents the strongest line in the image 

• Idea: Strong Lines in (x,y) space correspond to Maxima (or 
Peaks) in (p,0) space 

• Can be Used to extract other shapes, e.g. ellipses 


02/20/08 


Introduction to Image Processing 



SIMULATOR CURVE 


gradient magnitudes 


original 


gradient directions 


superposed lines 


Line Detection for Road Following 

(Autonomous Land Vehicle, AL V) 


Bootstrap Algorithm: 

1. Sobel Edge Detection 

2. Segment Orientation 
Histogram 

3. Create Magnitudes 
Images for each 
“Orientation Region” 

4. For each “Orientation 
Image”, Compute 
Hough Transform => 
Strongest Line 

5. Label All Lines 
According to World 

Model 
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Line Detection for Road Following 

(Autonomous Land Vehicle, AL V) 


Feedforward Algorithm: 

Strongest Orientation 
Known from Bootstrap Step 

l.Sobel Edge Detection 
2.Segment Orientation 
Histogram 

3. Create Magnitudes Images 
for Strongest “Orientation 
Region” from Bootstrap 

4. For Strongest “Orientation 
Image”, Compute Hough 
Transform => Strongest 
Line 



FEED-FORWARD PROCESSING 




a spas,; 




SUCCESSIVE STEPS 
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Region- Based Segmentation 

• Methods 

o Region Growing 
o Region Splitting 
o Split and Merge 
o Relaxation 
o Watershed Method 
o Means Cut 
o etc. 

• Parameters 

o Mean and Variance 
o Edge Magnitudes 
o Texture 

S Cooccurrence Matrix 
S Textural Edgeness 
S Filter Banks 

S Local Spatial Frequency Analysis, Gabor Filters and Wavelets 
S Mathematical Morphology 

o etc. 
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What is Texture ? 

• (Rosenfeld and Kak) “Visual Textures are 
complex visual patterns composed of entities, or 
subpattems (textons), that have characteristic 
brightnesses, color, slopes, sizes, etc. Thus a 
texture can be regarded as a similarity 
grouping.” 

• (Duraiswami) “Texture is something that repeats 
with variation”. 

• Some Texture Measures: 

- Co-occurrence Matrix: Statistics Computed 
from Distribution of Gray Levels across the 
Image at Different Orientations 

- “Filter Banks”: Filtering Image with 
Various Linear Filters Corresponding to 
Multiple Patterns at Various Scales, e.g. 

Weighted of Gaussians at Different Scales 

- Gabor or Wavelets Filters, Steerable 
Pyramid (Simoncelli): Provide Local Spatial 
Frequency Analysis 

Introduction to Image Processing 
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Region- Based Segmentation 

• Iterative Region Growing 

o First Iteration: Each pixel represents an individual region 

o Next Iteration: Regions are merged if the criterion for merging is 
satisfied (e.g., the variance of the pixel intensities in the merged region 
is below a given threshold) 

- Merged regions can be adjacent or not 

- One or several merges can happen at each iteration 

o Iterate until no more possible merging or until stopping criterion is 
satisfied, e.g., a minimum number of regions has been reached. 

o Successive Iterations can be represented by a tree structure where 
{Root: Complete Image; Leaves: Individual Pixels; Branches: 
Relations between Regions and Subregions} 

• Remarks: 

o Iterative Region Splitting 

- Reverse process starting with entire image as one region 

o Split and Merge 

- Iterative Succession of splitting and growing regions based on separate 
criteria for splitting and merging 
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Region- Based Segmentation 

Road Following Results 



BENDING ROAD 



Original 

Segmentation Line Segments 
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Region- Based Segmentation 

Landsat Thematic Mapper Segmentation 
(James Tilton/NASA GSFC) 
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Combining Regions and Edges 

Landsat Thematic Mapper Segmentation 


MasPar Application 


New Edge Image 
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Extract Iteration Contours 

1 

Compute Distance Threshold 

Find Best Iteration 

1 
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Image Modeling and 
Understanding 


02 / 20/08 


Introduction to Image Processing 


Understanding Images 

• A-Priori Knowledge 

- Domain Knowledge 

o Medical 

- Radiology or Cytology, ... If Cytology: Blood Cells, Cancer Cells, . . . 

o Remote Sensing 

- Space Science or Earth Science. If Earth Science: Agriculture, Change 
Detection (e.g., forest monitoring), Invasive Species, ... 

- World Model or “Ground Truth” 

• Based on “Low-Level and Intermediate-Level Processing” 

o Pixel Classification, Image Features, Grouping of Image Features, . . . 

• “High-Level Processing”: Image Understanding or Computer 
Vision/ Artificial Intelligence Techniques 

- Decision Trees, Knowledge-Based Systems, Expert Systems, 
Intelligent Agents 

o Object Recognition (e.g., Crater, Boulder, Rock Detection, . . .) 
o Region Labeling (e.g., Trees, Water, Road, Buildings, . . .) 
o 3D World Modeling (e.g., Pose Estimation for AR&D, . . .) 
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Blood Platelets Classification 

• A-Priori Knowledge 

- Blood Platelets Recognition after Freezing 

- Functional level related to Morphology and Texture 

• Image Understanding 

- Classification into 7 Classes based on geometric shapes, geometric and texture measurements, 
“Lenticular”, “Lenticular with Pseudopods”, “Circular”, “Circular with Pseudopods”, “Capping”, 
“Aggregate”, “Artefact” 



• Measurements: 

- Perimeter 

- Surface 

- Minimal Distance from 
Gravity Center to Contour 

- Maximal Distance from 
Gravity Center to Contour 

- Circularity Measure 

- Elongation Measures 

• Classification by Decision 
Tree and Rule-Based System 
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• A-Priori Knowledge 

- Road Networks 

- “Pencil of Lines” 
converging to a 
Vanishing Point 

• Image 
Understanding 

- Classification of 
Lines into 

• “Left Road” 

• “Left Shoulder” 

• “Right Road” 

• “Right Shoulder” 

• “Center Road” 

• “Discarded” 


Road Following 
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Additional Reading 

BOOKS: 

• A. Rosenfeld and A.C. Kak, “Digital Image Processing, ’’Academic Press, 1982. 

• D.A. Forsyth and J. Ponce, “Computer Vision: A Modem Approach,” Prentice 
Hall, 2003. 

• R.O. Duda and P.E. Hart, “Pattern Classification and Scene Analysis,” John 
Wiley & Sons, 1973. 

• B. Jahne, “Digital Image Processing: Concepts, Algorithms and Scientific 
Applications,” Springer- Verlag, 1991. 

• T.M. Lillesand and R.W. Kiefer, “Remote Sensing and Image Interpretation,” 
John Wiley & Sons, Inc., 1987. 

• J.G. Moik, “Digital Processing of Remotely Sensed Images,” NASA Technical 
Report SP-431, 1980. 

• P.H. Swain and S.M. Davis, “Remote Sensing: The Quantitative Approach,” 
McGraw Hill, 1978. 

ON LINE: 

• http : //www.umiacs .umd. edu/~ramani/cmsc42 6/ index.html 

• http://www.cs.cmu.edu/afs/cs/proiect/cil/ftp/html/vision.html 

• http://www.dai.ed.ac.uk/CVonline/ 
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